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ABSTRACT 

The process used in the design and evaluation of 
modules of instruction with the PLATO IV Computer System for stimulus 
display and response recording is described. Steps in the 
instructional design process are listed as problem identification and 
task analysis, identification of entry characteristics, development 
of performance objectives, development of evaluation instruments, 
determination of instructional sequence, design of instructional 
components, and production of instructional materials. Evaluation is 
discussed in terms of product, evaluation, process evaluation, and 
system effectiveness. (SK) 
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IMSTRUCTIOHAL DESIGH FOR THE FLORIDA PLATO PROJECT - 
PROCESS AnO EVALUATION 



William A. Broderick 
Center for Educational Design 
• Florida State University 
Tallahassee, Florida 32306 

It is the purpose of this paper to describe the process used In the 
Jesign and evaluation of modules of instruction v/ith the PLATO IV Computer 
System available for stimulus display and response recording. It is impor- 
tant to note that the approach is one of considering PLATO as one of several 
alternative delivery systems and not one of designing instruction to fit 
PLATO. This point wiV: be elaborated later fn the discussion on media 
selection. 

Instructional Design Process 
The process of instructional design in the PLATO project is one best 
described as a systems approach. Figure 1 is a schematic representation 
of the particular design model used in this project. This model describes 
the step-by-step procedures which have been used successfully in the design 
of programmed instruction materials, computer-assisted instruction courses, 
computer-managed instruction courses, and instructional activities using a 
variety of media. 

Problem Ide ntification and Task Aftalysis 

The first step in this model is to identify the instructional problem 
for which the materials are being designed. It is important to separate 
symptoms from problems so that the problem can be stated clearly and 
unambiguously. The problem statement should indicate clearly what the end 
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product for the learner is expected to be. From this information one can 
proceed to task-analyze the desired performance in order to identify its 
component parts and the relationships among them. The process of task 
analysis is often an iterative one which is closely related to designing 
the assessment of performance. The process may include content analysis, 
test item v/riting, task identification and then revisions to the point that 
one is satisfied with the inclusiveness and clarity of the task statements. 

Identif icati on of Entry Characteristics 

The next step in the model is to identify the entry characteristics of 
students. This step requires the instructional designer to consider the 
knowledge, skills, and attitudes that the learners bring to the particular 
learning situation. By considering the characteristics of the learner two 
types of errors can be avoided. The situation can be avoided where too much 
is expected of the learner, i.e., assumptions are made that he has skills 
which in fact he does not have^ and the situation can be avoided where the 
learner is being taught things he already knows. The characteristics of 
learners as a group should also be taken into consideration. Such factors 
as the social situation, the competitive or cooperative spirit and the out 
of school support or obstacles to learning should be considered. 

D evelopm ent of Performance Objectives 

The fourth step in the model is the development of performance objectives. 
Performance objectives are clear statements of the outcomes of the instruction 
in terms of v^hat the learners v/ill be expected to do. The development of 
these objectives is based on the results of the task analysis and the 
learner analysis. The exact form which these statements of objectives takes 



varies from one type of learning to another and may even vary fron one 
instructional design team to another. 

One form v/hich the objectives may take is that proposed by Mager who 
argues that if behavioral objectives are to be of real value they must 
include three characteristics. These characteristics are that they: 

(1) indicate the conditions under which the performance will take place; 

(2) indicate v;hat the performance is and (3) indicate the minimum level of 
acceptable performance. 

Some instructional designers prefer to state performance objectives in 
other ways such as in domains. A domain consists of a subset of knov/ledge, 
skill s» understandings or attitudes where the essential attributes of the 
content which the learner is to acquire, and behavior which will demonstrate 
such acquisition, are carefully described. Domcins are an attempt to make 
clear the instructional intent without being overly precise. 

The form of the objectives should not get in the way of the communication 
betv/een instructor and learner. If v/hat is expected is clearly communicated, 
the form is not of great significance. * 

Develop Evaluation Instru ments 

The next step in the model suggests that before the development of 
the instructional activities begins, evaluation instruments must be developed* 
In the development of a particular set of instructional materials, it is not 
likely that a unique set of evaluation instruments will be produced. Rather, 
the more general types of instruments discussed in a following section of 
the paper will be adapted for usa in a particular instructional setting. 

It v/ill be necessary^ hov/ever, to develop a unique set of instruments 
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for assessing learning outcomes since they will relate to the particular 
content of the lesson • These instruments must be designed to assess the 
performances identified in the objectives. The assessment instruments are 
not limited to paper and pencil tests but may use human observation and a 
variety of devices for stimulus presentation and response recording. The 
crucial factor is that the assessments are valid measures of attainment of the 
skills, knowledge, and attitudes which are the objectives of the instruction. 
The development of these instruments is an iterative process and both test 
items and objectives may be modified as lack of congruence betvveen them 
is discovered. The revision process should increase the relevance of the 
objectives and the validity of the test items. 

Determine Ins tructi ona l Sequence 

The sequence of instruction is considered in the sixth step of the 
model. The most effective sequence (if there is one) for presenting the 
instructional materials is determined by utilizing the ordered relation- 
ships between the subskills revealed by the task analysis procedure- 
For example 5 if the subskills are hierarchically related to each other, it 
is important that the instruction be designed to build one skill upon 
another until the terminal performance Is achieved. In other cases, the 
most effective sequence may be determined by the student who chooses 
from a selection of alternative sequences available to him. In all 
instructional situations s the instructional designer must be aware of 
the need for the sequencing of instruction, v;hether the sequence is 
linear and required cf all students^ or the sequence is flexible and can 
be different depending upon student choice. 
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Design Instructional Components 

In designing th^^ instructional components within various courses of 
the PLATO project the assumption is made that the materials will be used in 
a largely learner-paced system. The decisions to be made in designing each 
component begin v/ith determining the specific instructional events needed 
for the learner to accomplish the objective. For each instructional event, 
the design team identifies the type of stimuli to be presented to the student, 
the type of response to be required of the student and the type of response 
analysis capability required in order to make subsequent instructional 
decisions. 

Paul flerrilTs fledia Selection Matrix shov/n in Figure 2 is then used 
as a guide in selecting media (fterriT! 1975). The media included in the 
matrix are those available to F.S.U. faculty. Included in video are motion 
pictures as well as television. 

The cells of the matrix contain the letters G, F, P, or are left blank. 
If a cell contains +he letter "G" the media category listed at the top of the 
column is considered to have the capability and does a "good" job of pre- 
senting the stimuli, recording the response, or analyzing the response 
listed at the left of the row. The letter "F" indicates that the medium does 
a fair job, while the letter "P" means that the medium does a poor job. A 
blank cell indicates that the medium does not have the required capability. 

An example of the process may be useful at this point. An objective 
might be to have students be able to identify Gothic architecture - a concept 
acquisition task. The design team 

(1) selects pictures, line drav/ings and spoken v/ords as the stimuli; 
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(2) determines that student selective responses should be recorded; and 

(3) decides that the responses should be analyzed and instructional decisions 
made as a consequence of the analysis. 

Using the fledia Selection :iatrix one detannines that to satisfy all of these 
requirements will necessitate the use of PLATO or a combination of other 
media including slide projection, audio-tape, and a CRT computer system. 
This combination of media can be considered a delivery system which is in 
essence what PLATO is. The choice of a medium of instruction involves 
additional consideration of the system in which that medium will be delivered. 
For example* the television medium can be delivered live by broadcast or 
closed-circuit; or it can be recorded and delivered in a reel-to-reel, or 
cassette or soon with a disc system. 

Since PLATO has such media capabilities as audio, still pictures, 
graphics and visual words as well as the ability to input responses by 
typev/riter keyboard or touch panel, it is very tempting to use PLATO v/hen 
less expensive and more accessi'^le systems would be just as effective and 
efficient. Paul Merrill points out that PLATO'S unique combination of 
stimulus presentation and response analysis capabilities make it an appro- 
priate medium when the learning activity requires any of the following: 

1. Analysis of complex constructed responses (Natural language 
processing); 

2. Analysis of student responses and a coordinated random access 
presentation of several stimuli such as still pictures, audio 
sounds, and visual words, 

3. Analysis of student responses and the presentation of special 
symbols (non alphanumeric)-: 
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4, Analysis of student responses and the presentation of graphic 
displays (drawings, figures, charts, diagrams, graphs, etc.); or 

5. Analysis of student responses and the presentation of animated 
displ ays. 

PLATO can be used in many other ways such as displaying the text of 
a book, presently a slide/tape mo^ Oe, or simulating a programmed text. 
However, there are less expensive media which could be used for those 
applications. 

Produce Instructional Tfeiterials 

The process of actually producing the instructional materials will 
vary greatly from one medium to anot>.er and from one delivery system to another. 
The basic requirement in the process, no matter what the medium or delivery 
system, is that the content specialist be able to communicate his ideas 
clearly to the ultimate producer. 

If the PLATO system is to be used, a set of instructions to the computer 
must be v/ritten in a language called TUTOR. The content faculty member 
will describe his ideas in a series of cormands which are stated in terms 
of their purpose or function such as those commands which describe what is 
to be presented or displayed' on the plasma panel or screen. 

If a video lesson is to be produced, a script must be developed which 
Includes not only all of the narrative^, but also identifies each visual or 
visual sequence, the background music, if any, and any other information 
that the producer/director needs in completing the video tape. 

Once the materials or at least a prototype of the materials are 
produced, the formative evaluation of the products can begin. 
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Evaluation 

Evaluation is part of most if not all instructional development models, 
and the commitnent to doing something with the data collected is usually 
represented on a flov-chart by a series of lines and arrows labeled revision 
cycle v/hich may point to all steps of the module* You will note in Figure 1 
that the flowchart used in the PIATO Project is of that type* 

Evaluation in the PLATO developnent project will focus on representative 
parts of the process since extensive data collection is not possible for 
every project at all points indicated on the flov.»chart* Data are to be 
collected, hov/ever, on a systematic hesis for evaluation of process, of 
products and o? system effectiveness for use in formative and summative 
types of decisions. There v/ill be some overlap of data from one evaluation 
to another since these are interrelated rather than discrete entities. 
The relationship of the focus of the valuation to the type of decision is 
shovm in Figure 3. The types of data and decision alternatives in the 
cells of Fiqure 3 are meant to be examples rather than a listing of all 
possibilities. 

Process Evaluation 

One of the underlying goals of process evaluation is to determine 
whether procedures that were intended ac^-ually occurred, and if they did, 
what the observed consequences vjere. The basic met^'^d of process 
evaluation is accurate description and documentation. On-going description 
of activities and person-to-porson interactions serves to monitor processes 
and perinits the verification of events that were intended as well as the 
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identification of problems or processes that were unintended. Identifica- 
tion of problems^ if done on a timely basis, should lead to corrective 
measures. Judgments regarding the efficacy of procedures also need to be 
obtained. 

flethods for conducting the process evaluation include a thorough 
documentation and description of activities and procedures by the project 
staff. These data will be supplemented with information obtained through 
interviews with content faculty, designers, programmers, and anyone else 
involved in the instructional development process. In addition to providing 
descriptive information, the participants will also be asked to share their 
judgments regarding the efficacy of procedures. Judgments will be recorded 
on standardized scales. The final areas of information sought for process 
evaluation will include identification of problems and recormiendations 
for dealing with them. 

As seen in Figure 3 the data collected v;ill serve both formative and 
sunmative types of decisions with some of the same data used for both 
decision types. 

Product Evaluation 

Among the concerns in the overall evaluation design is an assessment 
of PLATO lessons used, their impact on learning, and impact on attitudes 
toward instructional use of the computer. Although there will still be a 
reliance on judgment, the methods used for this part of the study will 
emphasize quasi -experimental evaluation designs and assessment based on 
standardized, objective measures wherever possible. There will also be 
Some overlap v/ith areas covered by the process evaluation. The sources of 
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data v/ill include a content faculty review panel and a number of learners 
in both one-to-one and group settings. 

Standardized forms will be used to record faculty reviev-/er's and 
learner's judgments. Uhen appropriate responses will be recorded on 
5-point Likert-type rating scales. Dimensions on vihich lessons viill be 
judged would include at least the following: 
I nstructional Intent 
Clarity of purpose 
Clarity of instructional objectives 

Appropriateness of lesson difficulty for intended FSU students 
Content Validity 

Representativeness of information presented 

Representativeness of terminology and notation used 

Accuracy of information presented 
Instructional Logic 

Adequacy of sequencing 

Consistency of logic inherent in the subject matter 
Lesson Format and Text 

Attractiveness of lesson format 

Appropriateness of vocabulary 

Tone of feedback statements 
PJtSr?^. of i s ion Neede if L esson Is Adopted 
Overall Assessment of Lesson Quality 

An ir-ort'..^';t olo'Tjpt of the pro.iuct eval iation is the assc^ssitii.nt of 
learning outcomes to determine v/hether or not the relative advantage 
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presumed for PLATO does in fact obtain. Tho relative advantage of PLATO 
v/ould be verified if: (1) students using PLATO demonstrate equivalent 
levels of achievement in less time v/hen compared to students in a comparable 
nonPLATO course, or (2) students using PLATO demonstrate higher levels of 
achievement in the same or less time when compared to students in a 
comparable nonPLATO course. 

Determining relative advantage v/ill require two types of information: 
(1) time spent on the course; (2) achievement in relation to course 
objectives* iioreover, quasi -experimental designs will be used in order 
to make comparisons between PLATO and nonPLATO groups. 

Time spent on the course will be estimated from logs which PLATO and 
nonPLATO students will be required to keep and hand in on a regular basis. 
Achievement will be assessed using tests and procedures being used to evaluate 
students in the respective courses. If there is a difference in measures 
between PLATO and nonPLATO courses in the same content area, then both 
sets of tests vnll be given to each group to see if there is any overall 
superiority irrespective of the measures used. 

The ability to exert strict random assignment is compromised in the 
natural setting of the college classroom^ hence our reliance on quasi- 
experimental research designs. !^lot being able to meet the requirement of 
complete random assignment, vje vnll statistically adjust for group 
differences in aptitude. This approach is not without its problems as has 
been pointed out in the work of Lord (19579 1969). In spite of the 
shortcomings of adjusting for group differences statistically^ there 
appears to be no other more suitable alternative. In light of this a PLATO 
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and nonPLATO groups will he compared using a one-way nonstandard analysis of 
covariance using Florida 12th Grade Test score as covariate. This procedure 
permits the comparison of two groups along selected points of an aptitude 
continuum. In this way, observed main effects and interactions can be 
analyzed statistically after initial adjustment for group differences* 

Assessment of achievement in each academic area will be approached 
as follows. Achievement measures used for PLATO and nonPLATO courses 
will be given to both the PLATO and nonPLATO groups. Care will be taken 
to classify test items into groups which both the experimental and 
nonexperimental courses share in cormion. These common items will forn the 
basis for the least biased comparison. Results based on test items not 
common to both courses will be analyzed separately to assess overall 
effectiveness of learning and to make possible comparisons between groups. 

System Effectiveness 

In an instructional development system the whole is quite likely 
greater than the sum of its par^s. To be able to identify the extent to 
Vihich the intents of the entire system are accomplished, one focus of 
the evaluation will be on system effectiveness. This focus will include 
collecting data on the management system, the training system, as well as 
the process and products of instructional design. 

By systematically collecting data and relating them to intents it is 
hoped that meaningful revisions in the processes, the products, and the 
system can be accomplished and that as effectiveness and efficiency are 
improved, the accomplishments can be documented. 
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Sunmary 

The PLATO project at FSU is one which the instructional development 
effort is placing heavy emphasis on systematic design and evaluation. By 
clearly identifying the objectives and basing assessments of student perfor 
mance on these objectives, the effectiveness of products and processes can 
be more carefully evaluate-J. 



References 



Criggs, L. J., Handbook of pr ocedures for the design of instruction . 
Hashington: 'American institutes for Research^ 1970. 

Ganne, R. ii. an'J Driaqs, L. J., Principles of instr uctiona l d esign , 
flew York: Hoit, Rinehart and Ihnston/ Ts74.*' 

Lord, F. A paradox in the interpretation of group comparisons. 
Psy chological Dulleti n, 1967, G3, 304-305. 

Lord, F. Statistical adjustments when comparing pre-existing 
groups. Psychological Gulletin , 1969, 72 ^ 336-337. 

[lerrill, P. F., PLATO training program syllabus. Tallahassee, Fla.: 
Center for Educational Design, 1975. 

Stake, R. The countenance o^ educational evaluation. Teachers 
college Record , 1?67, GO. 

Stakenas, R. S., A test of the transferability of PLATO courseware, 
proposal submitted to the flational Science Foundation by 
Florida State University, narch. 1975. 

Towle, !!. J., The process of instructional development in the Center 
for Educational Design, report to the Florida Department of 
Education. Tallahassee, Fla.: Center for Educational Design, 
November, 1974. 



ERIC 



18 



